Multi-/hyperspectral two-dimensional image processing

ABSTRACT

There is provided an apparatus (100) comprising one or more processors (102) configured to acquire a multi-/hyperspectral two-dimensional image of an object at respective wavelengths. For at least one pixel of the image corresponding to a first point on an object surface, a set of intensity values for said at least one pixel is compared to a characteristic curve to determine a similarity measure. A first angle of the first point is estimated from the similarity measure or a correction is applied to the image at the first point using the similarity measure. The characteristic curve is a difference between a spectrum of at least one second point on the object surface at a second angle with respect to a plane of the image and a spectrum of at least one third point on the object surface at a third angle with respect to the plane of the image.

FIELD OF THE INVENTION

The disclosure relates to methods and apparatus for use inmulti-/hyperspectral two-dimensional image processing.

BACKGROUND OF THE INVENTION

Traditional RGB cameras capture visible light using only three channels:a red, green and blue channel. Multi-/hyperspectral cameras capture theelectromagnetic spectrum, both visible and invisible to the human eye,in a larger number of wavelengths (typically around 10 wavelengths formulti-spectral imaging or more than 100 wavelengths for hyperspectralimaging). As such, these multi-/hyperspectral cameras can revealproperties of an imaged object that are impossible to observe by thehuman eye.

Particularly, in the field of skin imaging, such multi-/hyperspectralcameras may be employed to estimate the concentration of chromophorespresent in the skin (e.g. melanin, carotenoids, water levels, lipids,etc.), which is not possible using traditional RGB cameras. Theestimated concentration of chromophores in the skin can provideinformation about the health of the skin but also more generally can beindicative of lifestyle or systemic health. It is especially interestingto process multi-/hyperspectral images of large surfaces of skin, suchas the human face. Tracking multi-/hyperspectral images over time mayreveal particular local changes in chromophore concentrations that maybe attributed to e.g. lifestyle changes. The process of estimating thechromophore concentrations from a spectrum is referred to as spectraldecomposition. As an example, FIG. 1 shows calibrated hyperspectralimages acquired at six different wavelengths of light; 448 nm, 494 nm,610 nm, 669 nm, 812 nm and 869 nm. In FIG. 1, the pixel location isprovided by the x and y axes, and the greyscale illustrates reflectance.The whiter the pixel, the higher the intensity and thus the higher thereflectance.

In order to allow for a proper estimation of the chromophores of acurved surface (e.g. the skin, such as the human face), the angle atwhich the curved surface is positioned with respect to the camera needsto be known. This angle is needed as light is absorbed, scattered andreflected in the different layers of the curved surface leading to astrong angular dependency. To arrive at such angular data, one or morethree-dimensional (3D) cameras are typically installed, e.g.Time-of-Flight cameras, in order to register a 3D image and derive anangular map.

However, the use of 3D cameras in a multi-/hyperspectral setup iscumbersome and costly, since an additional camera is required and atwo-dimensional (2D) multi-/hyperspectral image needs to be combinedwith the 3D image, which requires the use of advanced image registrationtechniques and accurate mapping of the 2D image with the 3D image. Theselimitations are especially apparent when moving towards consumer type ofrealizations. The use of one or more additional cameras prevent a smallform factor.

SUMMARY OF THE INVENTION

As noted above, the limitations with existing techniques is that, inorder to acquire information suitable for observing properties of anobject (e.g. the concentration of chromophores in skin), an additionalcamera is required and a 2D multi-/hyperspectral image needs to becombined with a 3D image, which requires advanced image registrationtechniques and accurate mapping of the 2D image with the 3D image. Itwould thus be valuable to have an improvement aimed at addressing theselimitations.

Therefore, according to a first aspect, there is provided an apparatusfor estimating a first angle of a first point on a surface of an objectfrom a multi-/hyperspectral two-dimensional image of the object atrespective wavelengths or applying a correction to themulti-/hyperspectral two-dimensional image. The apparatus comprises oneor more processors configured to acquire a multi-/hyperspectraltwo-dimensional image of an object at respective wavelengths. Themulti-/hyperspectral two-dimensional image at the respective wavelengthsis formed of a plurality of pixels. Each pixel has a set of intensityvalues corresponding to light intensity values for each of a pluralityof wavelengths of light. The one or more processors are configured to,for at least one pixel of the plurality of pixels corresponding to afirst point on a surface of the object, compare the set of intensityvalues for said at least one pixel to a characteristic curve for theobject to determine a measure of similarity of the set of intensityvalues to the obtained characteristic curve. The one or more processorsare configured to estimate a first angle of the first point on thesurface of the object corresponding to said at least one pixel from thedetermined measure of similarity or apply a correction to themulti-/hyperspectral two-dimensional image at the first point on thesurface of the object using the determined measure of similarity. Thecharacteristic curve is indicative of a difference between a spectrum ofat least one second point on the surface of the object at a second anglewith respect to a plane of the image and a spectrum of at least onethird point on the surface of the object at a third angle with respectto the plane of the image.

In some embodiments, the first angle may be with respect to the plane ofthe image. In some embodiments, the characteristic curve maycharacterize how the object reflects or absorbs light as a function ofangle.

In some embodiments, the characteristic curve may be indicative of adifference between an average spectrum of at least two second points onthe surface of the object at the second angle with respect to the planeof the image and a spectrum of at least two third points on the surfaceof the object at the third angle with respect to the plane of the image.

In some embodiments, the third angle may be a known angle that isdifferent from or substantially different from the second angle. In someembodiments, the second angle may be about 0 degrees and/or the thirdangle may be an angle in a range from 45 to 90 degrees.

In some embodiments, the at least one second point on the surface of theobject may comprise at least one brightest point on the surface of theobject and/or the at least one third point on the surface of the objectmay comprise at least one dimmest point on the surface of the object.

In some embodiments, the at least one second point on the surface of theobject may be identified by using landmark detection to detect at leastone second point on the surface of the object at the second angle withrespect to the plane of the image and/or the at least one third point onthe surface of the object may be identified by using landmark detectionto detect at least one third point on the surface of the object at thethird angle with respect to the plane of the image.

In some embodiments, the characteristic curve may be predetermined usingat least one other multi-/hyperspectral two-dimensional image of thesame type of object or the characteristic curve may be determined usingthe multi-/hyperspectral two-dimensional image of the object.

In some embodiments, the one or more processors may be configured to,for at least one pixel of the plurality of pixels corresponding to atleast one other first point on the surface of the object, compare theset of intensity values for said at least one pixel to thecharacteristic curve for the object to determine a measure of similarityof the set of intensity values to the obtained characteristic curve. Inthese embodiments, the one or more processors may be configured toestimate at least one other first angle of the at least one other firstpoint on the surface of the object corresponding to said at least onepixel from the determined measure of similarity and derive an angularmap comprising the estimated first angle and the estimated at least oneother first angle.

In some embodiments, the characteristic curve for the object maycomprise (or be selected from) a set of characteristic curves for arespective set of second and third angles.

In some embodiments, the spectrum of the at least one second point maycomprise a reflectance spectrum indicative of the portion of lightreflected from the object at the at least one second point on thesurface of the object at the second angle with respect to the plane ofthe image or an absorbance spectrum indicative of the portion of lightabsorbed by the object at the at least one second point on the surfaceof the object at the second angle with respect to the plane of theimage, and/or the spectrum of the at least one third point may comprisea reflectance spectrum indicative of the portion of light reflected fromthe object at the at least one third point on the surface of the objectat the third angle with respect to the plane of the image or anabsorbance spectrum indicative of the portion of light absorbed by theobject by the object at the at least one third point on the surface ofthe object at the third angle with respect to the plane of the image.

In some embodiments, the object may be skin and the one or moreprocessors may be configured to determine a concentration ofchromophores in the skin from the multi-/hyperspectral two-dimensionalimage of the skin using the estimated first angle or from themulti-/hyperspectral two-dimensional image with the correction applied.

According to a second aspect, there is provided a method for estimatinga first angle of a first point on a surface of an object from amulti-/hyperspectral two-dimensional image of the object at respectivewavelengths or applying a correction to the multi-/hyperspectraltwo-dimensional image. The method comprises acquiring amulti-/hyperspectral two-dimensional image of an object at respectivewavelengths, wherein the multi-/hyperspectral two-dimensional image atthe respective wavelengths is formed of a plurality of pixels, eachpixel having a set of intensity values corresponding to light intensityvalues for each of a plurality of wavelengths of light. The methodcomprises, for at least one pixel of the plurality of pixelscorresponding to a first point on a surface of the object, comparing theset of intensity values for said at least one pixel to a characteristiccurve for the object to determine a measure of similarity of the set ofintensity values to the obtained characteristic curve. The methodcomprises estimating a first angle of the first point on the surface ofthe object corresponding to said at least one pixel from the determinedmeasure of similarity or apply a correction to the multi-/hyperspectraltwo-dimensional image at the first point on the surface of the objectusing the determined measure of similarity. The characteristic curve isindicative of a difference between a spectrum of at least one secondpoint on the surface of the object at a second angle with respect to aplane of the image and a spectrum of at least one third point on thesurface of the object at a third angle with respect to the plane of theimage.

In some embodiments, the first angle may be with respect to the plane ofthe image. In some embodiments, the characteristic curve maycharacterize how the object reflects or absorbs light as a function ofangle. In some embodiments, the third angle may be a known angle that isdifferent from or substantially different from the second angle.

According to a third aspect, there is provided an apparatus fordetermining a characteristic curve for use in estimating a first angleof a first point on a surface of an object from a multi-/hyperspectraltwo-dimensional image of the object at respective wavelengths or inapplying a correction to the multi-/hyperspectral two-dimensional image.The apparatus comprises one or more processors configured to acquire afirst spectrum of at least one second point on the surface of the objectat a second angle with respect to a plane of the image, acquire a secondspectrum of at least one third point on the surface of the object at athird angle with respect to the plane of the image, and determine thecharacteristic curve as a difference between the first spectrum and thesecond spectrum.

In some embodiments, the first angle may be with respect to the plane ofthe image. In some embodiments, the characteristic curve maycharacterize how the object reflects or absorbs light as a function ofangle. In some embodiments, the third angle may be a known angle that isdifferent from or substantially different from the second angle.

According to a fourth aspect, there is provided a method for determininga characteristic curve for use in estimating a first angle of a firstpoint on a surface of an object from a multi-/hyperspectraltwo-dimensional image of the object at respective wavelengths or inapplying a correction to the multi-/hyperspectral two-dimensional image.The method comprises acquiring a first spectrum of at least one secondpoint on the surface of the object at a second angle with respect to aplane of the image, acquiring a second spectrum of at least one thirdpoint on the surface of the object at a third angle with respect to theplane of the image, and determining the characteristic curve as adifference between the first spectrum and the second spectrum.

In some embodiments, the first angle may be with respect to the plane ofthe image. In some embodiments, the characteristic curve maycharacterize how the object reflects or absorbs light as a function ofangle. In some embodiments, the third angle may be a known angle that isdifferent from or substantially different from the second angle.

According to a fifth aspect, there is provided a computer programproduct comprising a computer readable medium. The computer readablemedium has a computer readable code embodied therein. The computerreadable code is configured such that, on execution by a suitablecomputer or processor, the computer or processor is caused to performthe method described earlier.

According to the aspects and embodiments described above, thelimitations of existing techniques are addressed. In particular,according to the above-described aspects and embodiments, the need foran additional camera and advanced image registration techniques forcombining a 2D multi-/hyperspectral image with the 3D image is in orderto acquire information suitable for observing properties of an object(e.g. the concentration of chromophores in skin) is overcome. A 3Dimaging modality in multi-/hyperspectral imaging is no longer required.According to the above-described aspects and embodiments, angularinformation can be reliably derived directly from themulti-/hyperspectral image without the use of additional camera signals.This is realized by (deriving and) exploiting a characteristic curve.Alternatively, the multi-/hyperspectral image is pre-processed using thecharacteristic curve to compensate for the effects of angle on thespectrum.

There is thus provided useful techniques for estimating an angle of apoint on a surface of an object from a multi-/hyperspectraltwo-dimensional image of the object at respective wavelengths orapplying a correction to the multi-/hyperspectral two-dimensional image.There is also provided a useful technique for determining acharacteristic curve for use in such an estimation or correction.

These and other aspects will be apparent from and elucidated withreference to the embodiment(s) described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments will now be described, by way of example only,with reference to the following drawings, in which:

FIG. 1 is an example of hyperspectral images acquired at six differentwavelengths of light;

FIG. 2 is a schematic illustration of an apparatus according to anembodiment;

FIG. 3 is a flow chart illustrating a method according to an embodiment;

FIG. 4 is a flow chart illustrating a method according to an embodiment;

FIG. 5 is an example schematic illustration of a hyperspectral imagetogether with a hyperspectral map of a slice from that image;

FIG. 6 is an example of a change in calibrated hyperspectral image as afunction of distance;

FIG. 7 is an example of a characteristic curve; and

FIG. 8 is an example of a derived angular map.

DETAILED DESCRIPTION OF THE EMBODIMENTS

As noted above, there is provided herein a technique for estimating anangle of a point on a surface of an object from a multi-/hyperspectraltwo-dimensional image of the object at respective wavelengths orapplying a correction to the multi-/hyperspectral two-dimensional image.Herein, an object may be any type of object. In some embodiments, theobject may be any object having a curved surface. In some embodiments,for example, the object may be skin or the (skin of the) face of asubject.

FIG. 2 illustrates an apparatus 100 for estimating a first angle of afirst point on a surface of an object from a multi-/hyperspectraltwo-dimensional image of the object at respective wavelengths orapplying a correction to the multi-/hyperspectral two-dimensional imageaccording to an embodiment. In some embodiments, the apparatus 100 maybe a device (e.g. a consumer device) or an attachment (or add-on) for adevice. A device can, for example, be a phone (e.g. a smart phone), atablet, or any other device. As illustrated in FIG. 2, the apparatus 100comprises one or more processors 102.

The one or more processors 102 can be implemented in numerous ways, withsoftware and/or hardware, to perform the various functions describedherein. In particular implementations, the one or more processors 102can comprise a plurality of software and/or hardware modules, eachconfigured to perform, or that are for performing, individual ormultiple steps of the method described herein. The one or moreprocessors 102 may comprise, for example, one or more microprocessors,one or more multi-core processors and/or one or more digital signalprocessors (DSPs), one or more processing units, and/or one or morecontrollers (e.g. one or more microcontrollers) that may be configuredor programmed (e.g. using software or computer program code) to performthe various functions described herein. The one or more processors 102may be implemented as a combination of dedicated hardware (e.g.amplifiers, pre-amplifiers, analog-to-digital convertors (ADCs) and/ordigital-to-analog convertors (DACs)) to perform some functions and oneor more processors (e.g. one or more programmed microprocessors, DSPsand associated circuitry) to perform other functions.

Briefly, the one or more processors 102 of the apparatus 100 areconfigured to acquire a multi-/hyperspectral two-dimensional image of anobject at respective wavelengths. The multi-/hyperspectraltwo-dimensional image at the respective wavelengths is formed of aplurality of pixels. Each pixel has a set of intensity valuescorresponding to light intensity values for each of a plurality ofwavelengths of light. The one or more processors 102 of the apparatus100 are also configured to, for at least one pixel of the plurality ofpixels corresponding to a first point on a surface of the object,compare the set of intensity values for said at least one pixel to acharacteristic curve for the object to determine a measure of similarityof the set of intensity values to the obtained characteristic curve. Theone or more processors 102 of the apparatus 100 are also configured toestimate a first angle of the first point on the surface of the objectcorresponding to said at least one pixel from the determined measure ofsimilarity or apply a correction to the multi-/hyperspectraltwo-dimensional image at the first point on the surface of the objectusing the determined measure of similarity.

The characteristic curve referred to herein is indicative of adifference between a spectrum of at least one second point on thesurface of the object at a second angle with respect to a plane of theimage and a spectrum of at least one third point on the surface of theobject at a third angle with respect to the plane of the image.

Herein, the spectrum of the at least one second point may comprise areflectance spectrum indicative of the portion of light reflected fromthe object at the at least one second point on the surface of the objectat the second angle with respect to the plane of the image or anabsorbance spectrum indicative of the portion of light absorbed by theobject at the at least one second point on the surface of the object atthe second angle with respect to the plane of the image. Alternativelyor in addition, herein, the spectrum of the at least one third point maycomprise a reflectance spectrum indicative of the portion of lightreflected from the object at the at least one third point on the surfaceof the object at the third angle with respect to the plane of the imageor an absorbance spectrum indicative of the portion of light absorbed bythe object at the at least one third point on the surface of the objectat the third angle with respect to the plane of the image. Herein, areflectance spectrum can generally be understood to mean a relativeamount of light reflected as a function of wavelength. Similarly,herein, an absorbance spectrum can generally be understood to mean arelative amount of light absorbed as a function of wavelength.

In some embodiments, the one or more processors 102 of the apparatus 100can be configured to acquire the multi-/hyperspectral two-dimensionalimage of the object at respective wavelengths from an imaging sensor104. The imaging sensor may, for example, be a camera or, morespecifically, a multi-/hyperspectral camera. As illustrated in FIG. 2,in some embodiments, the apparatus 100 may comprise the imaging sensor104. Alternatively or in addition, in some embodiments, the imagingsensor 104 may be external to (e.g. separate to or remote from) theapparatus 100. For example, another apparatus (or device, e.g. a capturedevice) may comprise the imaging sensor 104 according to someembodiments.

As illustrated in FIG. 2, in some embodiments, the apparatus 100 maycomprise at least one memory 106. Alternatively or in addition, in someembodiments, at least one memory 106 may be external to (e.g. separateto or remote from) the apparatus 100. For example, another apparatus maycomprise at least one memory 106 according to some embodiments. In someembodiments, a hospital database may comprise at least one memory 106,at least one memory 106 may be a cloud computing resource, or similar.The one or more processors 102 of the apparatus 100 may be configured tocommunicate with and/or connect to at least one memory 106. The at leastone memory 106 may comprise any type of non-transitory machine-readablemedium, such as cache or system memory including volatile andnon-volatile computer memory such as random access memory (RAM), staticRAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable ROM(PROM), erasable PROM (EPROM), and electrically erasable PROM (EEPROM).In some embodiments, at least one memory 106 can be configured to storeprogram code that can be executed by the one or more processors 102 ofthe apparatus 100 to cause the apparatus 100 to operate in the mannerdescribed herein.

Alternatively or in addition, at least one memory 106 can be configuredto store information required by or resulting from the method describedherein. For example, at least one memory 106 may be configured to storethe multi-/hyperspectral two-dimensional image of the object, thedetermined measure of similarity, the estimated first angle of the firstpoint on the surface of the object, the multi-/hyperspectraltwo-dimensional image of the object with the correction applied, or anyother information, or any combination of information, required by orresulting from the method described herein. The one or more processors102 of the apparatus 100 can be configured to control at least onememory 106 to store information required by or resulting from the methoddescribed herein.

As illustrated in FIG. 2, in some embodiments, the apparatus 100 maycomprise at least one user interface 108. Alternatively or in addition,in some embodiments, at least one user interface 108 may be external to(e.g. separate to or remote from) the apparatus 100. The one or moreprocessors 102 of the apparatus 100 may be configured to communicatewith and/or connect to at least one user interface 108. In someembodiments, one or more processors 102 of the apparatus 100 can beconfigured to control at least one user interface 108 to operate in themanner described herein.

A user interface 108 can be configured to render (or output, display, orprovide) information required by or resulting from the method describedherein. For example, in some embodiments, one or more user interfaces108 may be configured to render (or output, display, or provide) any oneor more of the multi-/hyperspectral two-dimensional image of the object,the determined measure of similarity, the estimated first angle of thefirst point on the surface of the object, the multi-/hyperspectraltwo-dimensional image of the object with the correction applied, or anyother information, or any combination of information, required by orresulting from the method described herein. Alternatively or inaddition, one or more user interfaces 108 can be configured to receive auser input. For example, one or more user interfaces 108 may allow auser to manually enter information or instructions, interact with and/orcontrol the apparatus 100. Thus, one or more user interfaces 108 may beany one or more user interfaces that enable the rendering (oroutputting, displaying, or providing) of information and/or enables auser to provide a user input.

The user interface 108 may comprise one or more components for this. Forexample, one or more user interfaces 108 may comprise one or moreswitches, one or more buttons, a keypad, a keyboard, a mouse, a displayor display screen, a graphical user interface (GUI) such as a touchscreen, an application (e.g. on a smart device such as a tablet, a smartphone, or any other smart device), or any other visual component, one ormore speakers, one or more microphones or any other audio component, oneor more lights (e.g. one or more light emitting diodes, LEDs), acomponent for providing tactile or haptic feedback (e.g. a vibrationfunction, or any other tactile feedback component), a smart device (e.g.a smart mirror, a tablet, a smart phone, a smart watch, or any othersmart device), or any other user interface, or combination of userinterfaces. In some embodiments, one or more user interfaces that arecontrolled to render information may be the same as one or more userinterfaces that enable the user to provide a user input.

As illustrated in FIG. 2, in some embodiments, the apparatus 100 maycomprise at least one communications interface (or communicationscircuitry) 110. Alternatively or in addition, in some embodiments, atleast one communications interface 110 may be external to (e.g. separateto or remote from) the apparatus 100. A communications interface 110 canbe for enabling the apparatus 100, or components of the apparatus 100(e.g. one or more processors 102, one or more sensors 104, one or morememories 106, one or more user interfaces 108 and/or any othercomponents of the apparatus 100), to communicate with and/or connect toeach other and/or one or more other components. For example, one or morecommunications interfaces 110 can be for enabling one or more processors102 of the apparatus 100 to communicate with and/or connect to one ormore sensors 104, one or more memories 106, one or more user interfaces108 and/or any other components of the apparatus 100.

A communications interface 110 may enable the apparatus 100, orcomponents of the apparatus 100, to communicate and/or connect in anysuitable way. For example, one or more communications interfaces 110 mayenable the apparatus 100, or components of the apparatus 100, tocommunicate and/or connect wirelessly, via a wired connection, or viaany other communication (or data transfer) mechanism. In some wirelessembodiments, for example, one or more communications interfaces 110 mayenable the apparatus 100, or components of the apparatus 100, to useradio frequency (RF), Bluetooth, or any other wireless communicationtechnology to communicate and/or connect.

FIG. 3 illustrates a method 200 for estimating a first angle of a firstpoint on a surface of an object from a multi-/hyperspectraltwo-dimensional image of the object at respective wavelengths orapplying a correction to the multi-/hyperspectral two-dimensional imageaccording to an embodiment. More specifically, FIG. 3 illustrates amethod 200 of operating the apparatus 100 described earlier withreference to FIG. 2 for estimating a first angle of a first point on asurface of an object from a multi-/hyperspectral two-dimensional imageof the object at respective wavelengths or applying a correction to themulti-/hyperspectral two-dimensional image.

The method 200 illustrated in FIG. 3 is a computer-implemented method.As described earlier with reference to FIG. 2, the apparatus 100comprises one or more processors 102. The method 200 illustrated in FIG.3 can generally be performed by or under the control of the one or moreprocessors 102 of the apparatus 100 described earlier with reference toFIG. 2.

With reference to FIG. 3, at block 202, a multi-/hyperspectraltwo-dimensional image of an object at respective wavelengths isacquired. As mentioned earlier, the multi-/hyperspectral two-dimensionalimage at the respective wavelengths is formed of a plurality of pixels.Each pixel having a set of intensity values corresponding to lightintensity values for each of a plurality of wavelengths of light.

At block 204 of FIG. 3, for at least one pixel of the plurality ofpixels corresponding to a first point on a surface of the object, theset of intensity values for said at least one pixel is compared to acharacteristic curve (or profile) for the object to determine a measureof similarity of the set of intensity values to the obtainedcharacteristic curve. As mentioned earlier, the characteristic curvereferred to herein is indicative of the difference between the spectrumof the at least one second point on the surface of the object at thesecond angle with respect to the plane of the image and the spectrum ofthe at least one third point on the surface of the object at the thirdangle with respect to the plane of the image. The characteristic curvecan be predetermined. In some embodiments, the characteristic curve maybe a predetermined characteristic curve stored in a memory (e.g. amemory 106 of the apparatus 100 or any other memory). This can beadvantageous since it reduces the computation complexity and avoids(manual) errors to improve accuracy.

The characteristic curve can be a curve that characterizes how theobject reflects or absorbs light as a function of angle. In this way, itis possible to establish the extent to which the characteristic curve ispresent (or visible) for each spatial location (x, y). The extent towhich the characteristic curve is present is a measure of the angle.

In more detail, in some embodiments, the characteristic curve cancharacterize how the object reflects (or absorbs) light as a function ofangle according to the following equation:

Reflectance[λ, α]=Reflectance[λ, 0]=α*characteristic_curve[λ].

where λ denotes wavelength and α denotes angle. Thus, if the reflectancespectrum of the object is measured under different angles, it can beobserved that this reflectance spectrum can be decomposed into thereflectance spectrum at an angle of 0 degrees and a part that linearlydepends on the angle. The extent to which the characteristic curve isavailable is then a measure of the angle. In terms of the correction tobe applied to the multi-/hyperspectral two-dimensional image, this cancomprise approximating the Reflectance[λ, 0] according to the aboveequation.

Thus, at block 204 of FIG. 3, for at least one pixel of the plurality ofpixels corresponding to the first point on the surface of the object,the set of intensity values for said at least one pixel is compared tothe characteristic curve for the object to determine the measure ofsimilarity of the set of intensity values to the characteristic curve.In some embodiments, the set of intensity values for said at least onepixel may be compared directly to the characteristic curve to determinethe measure of similarity. In other embodiments, one or more featuresmay be derived from the set of intensity values for said at least onepixel and the one or more derived features may be compared to one ormore corresponding features of the characteristic curve to determine themeasure of similarity. Examples of the one or more features include, butare not limited to, a slope of a curve, a standard deviation of a curve,etc.

Herein, the spectrum of the at least one second point on the surface ofthe object at the second angle with respect to the plane of the imagemay be referred to as the baseline spectrum. Also, herein, the spectrumof the at least one third point on the surface of the object at thethird angle with respect to the plane of the image may be referred to asthe edge spectrum. Thus, the characteristic curve can be defined as thedifference between the baseline spectrum and the edge spectrum.

In some embodiments, the measure of similarity that is determined atblock 204 of FIG. 3 may comprise a measure of correlation. In someembodiments, the measure of similarity (e.g. the measure of correlation)may be determined using regression analysis. In an example, linearregression (or correlation) can be employed for at least one pixel ofthe plurality of pixels or on a pixel-by-pixel basis, e.g. for every x,ypair of pixels. This can be represented using a vector notation asfollows:

a=(p ^(T) p)⁻¹ p ^(T) d,

where p is the characteristic curve (represented as a column vector), dis an input spectrum at position x,y (represented as a column vector)and a is the regression coefficient at position x,y, which establishesthe extent to which the characteristic curve is present in the inputspectrum. Thus, the regression coefficient a is the measure ofsimilarity in this example. The operator T denotes a transposition. Theinput spectrum at position x,y is the set of intensity values for saidat least one pixel described earlier, where the set of intensity valuescorrespond to the light intensity values for each of the plurality ofwavelengths λ of light.

In some embodiments, the input spectrum d may directly be the vectors=s[λ, x, y] for a given x,y coordinate and the plurality of wavelengthsλ. Alternatively, the input spectrum d may be baseline compensated bysubtracting the baseline spectrum, such that d[λ]=s[λ,x,y]−s_(baseline)[λ] for a given x,y coordinate for the plurality ofwavelengths λ. This can result in improved robustness.

Returning back to FIG. 3, at block 206, a first angle of the first pointon the surface of the object corresponding to said at least one pixel isestimated from the determined measure of similarity or a correction isapplied to the multi-/hyperspectral two-dimensional image at the firstpoint on the surface of the object using the determined measure ofsimilarity. The first angle can be with respect to the plane of theimage. Thus, angular information may be derived from themulti-/hyperspectral image by exploiting the characteristic curve or themulti-/hyperspectral image may be pre-processed using the characteristiccurve. In some embodiments where the regression coefficient a is themeasure of similarity, the regression coefficient a may be a directestimate of the first angle of the first point on the surface of theobject corresponding to said at least one pixel. For example, in someembodiments, the first angle of the first point on the surface of theobject corresponding to said at least one pixel may be estimated asfollows:

α=min(max(c·a, 0), π/2),

where c is a predetermined constant, and the min, max operator preventsangles being estimated outside the range [0, pi/2].

In some embodiments where the correction is applied to themulti-/hyperspectral two-dimensional image at the first point on thesurface of the object, the multi-/hyperspectral image can bepre-processed in order to compensate for the effects of angle. In someembodiments, the correction may comprise flattening the image. In someembodiments, pre-processing of the spectrum s[λ, x, y] for a givencoordinate x, y and the plurality of wavelengths λ, may be realized byestablishing the residual after regression, as follows:

s _(preprocessed)[λ, x, y]=s[λ, x, y]−a[x, y]·s _(characteristic)[λ].

In some embodiments, a set of characteristic curves may be stored in amemory (e.g. a memory 106 of the apparatus 100 or any other memory). Insome of these embodiments, at block 204 of FIG. 3, for at least onepixel of the plurality of pixels corresponding to the first point on thesurface of the object, the set of intensity values for said at least onepixel may be compared to each characteristic curve in the set ofcharacteristic curves to determine respective measures of similarity ofthe set of intensity values to the characteristic curves. Thecharacteristic curve in the set of characteristic curves that is mostsimilar to (i.e. best matches) the set of intensity values is selected.For example, a best fit technique may be used. In some embodiments, eachcharacteristic curve in the set of characteristic curves may be storedwith a corresponding angle, e.g. in the form of a look-up table. Thus,in these embodiments, at block 206 of FIG. 3, the first angle may bedetermined as the angle that is stored with the selected characteristiccurve. Alternatively, at block 206 of FIG. 3, the selectedcharacteristic curve may be used to correct the set of intensity values.

There is also provided herein an apparatus for determining acharacteristic curve for use in estimating a first angle of a firstpoint on a surface of an object from a multi-/hyperspectraltwo-dimensional image of the object at respective wavelengths asdescribed herein or in applying a correction to the multi-/hyperspectraltwo-dimensional image as described herein. The apparatus comprises oneor more processors as described earlier with reference to FIG. 2 and mayalso comprise any one or more of the other components of the apparatus100 described earlier with reference to FIG. 2.

In some embodiments, the apparatus for determining the characteristiccurve may be the same apparatus 100 as described earlier with referenceto FIG. 2. Thus, in some embodiments, the apparatus 100 describedearlier with reference to FIG. 2 can also be for determining thecharacteristic curve referred to herein. Alternatively, in someembodiments, the apparatus for determining the characteristic curvereferred to herein may be a different apparatus to the apparatus 100described earlier for estimating the first angle of the first point onthe surface of the object from a multi-/hyperspectral two-dimensionalimage of the object at respective wavelengths as described herein or inapplying the correction to the multi-/hyperspectral two-dimensionalimage as described herein.

Briefly, the one or more processors of the apparatus for determining thecharacteristic curve are configured to acquire the first spectrum of theat least one second point on the surface of the object at the secondangle with respect to the plane of the image, acquire the secondspectrum of the at least one third point on the surface of the object atthe third angle with respect to the plane of the image and determine thecharacteristic curve as the difference between the first spectrum andthe second spectrum.

FIG. 4 illustrates a method 300 for determining a characteristic curvefor use in estimating a first angle of a first point on a surface of anobject from a multi-/hyperspectral two-dimensional image of the objectat respective wavelengths or in applying a correction to themulti-/hyperspectral two-dimensional image according to an embodiment.More specifically, FIG. 4 illustrates a method 300 of operating theapparatus for determining a characteristic curve.

The method 300 illustrated in FIG. 4 is a computer-implemented method.As mentioned earlier, the apparatus for determining the characteristiccurve comprises one or more processors. The method 300 illustrated inFIG. 4 can generally be performed by or under the control of the one ormore processors of the apparatus.

With reference to FIG. 4, at block 302, the first spectrum of the atleast one second point on the surface of the object at the second anglewith respect to the plane of the image is acquired. At block 304 of FIG.4, the second spectrum of the at least one third point on the surface ofthe object at the third angle with respect to the plane of the image isacquired. At block 306 of FIG. 4, the characteristic curve is determinedas the difference between the first spectrum and the second spectrum.

In some embodiments, a method may comprise blocks 302 to 306 of FIG. 4and blocks 202 to 206 of FIG. 3. For example, the method may compriseblocks 302 to 306 of FIG. 4 followed by blocks 202 to 206 of FIG. 3.Alternatively, for example, the method may comprise block 202 of FIG. 3,followed by blocks 302 to 306 of FIG. 4, followed by blocks 204 to 206of FIG. 3, or any other suitable order of blocks.

As mentioned earlier, the characteristic curve referred to herein isindicative of the difference between the spectrum of the at least onesecond point on (e.g. the plane of) the surface of the object at thesecond angle with respect to the plane of the image and the spectrum ofthe at least one third point on (e.g. the plane of) the surface of theobject at the third angle with respect to the plane of the image.

The second angle referred to herein may instead be defined as a secondangle with respect to a plane of the optical lens of the imaging sensor.Similarly, the third angle referred to herein may instead be defined asa third angle with respect to a plane of the optical lens of the imagingsensor. In some embodiments, the characteristic curve may be indicativeof a difference between an average spectrum of at least two secondpoints on (e.g. the plane of) the surface of the object at the secondangle with respect to the plane of the image and a spectrum of at leasttwo third points on (e.g. the plane of) the surface of the object at thethird angle with respect to the plane of the image. In some embodiments,the third angle may be a known angle that is different from orsubstantially different from the second angle.

In some embodiments, the second angle referred to herein may be about 0degrees. That is, in some embodiments, the at least one second point onthe surface of the object may be at about 0 degrees with respect to (orparallel or substantially/approximately parallel to) the plane of theimage. Put differently, in some embodiments, the at least one secondpoint on the surface of the object may be at about 0 degrees withrespect to (or parallel or substantially/approximately parallel to) theplane of an optical lens of the imaging sensor. Thus, in someembodiments, the second angle may be such that (e.g. the plane of) thesurface of the object at the second point and the plane of the image (orthe plane of the optical lens of the imaging sensor) are parallel orsubstantially/approximately parallel. The baseline spectrum mentionedearlier can thus be a representative spectrum for location(s) where theplane of the image (or the plane of the optical lens of the imagingsensor) is parallel or substantially/approximately parallel to (e.g. theplane of) the surface of the object.

Alternatively or in addition, in some embodiments, the third anglereferred to herein may be an angle in a range from 45 to 90 degrees, forexample an angle in a range from 50 to 85 degrees, for example an anglein a range from 55 to 80 degrees, for example an angle in a range from60 to 75 degrees. For example, in some embodiments, the third anglereferred to herein may be an angle selected from 45 degrees, 50 degrees,55 degrees, 60 degrees, 65 degrees, 70 degrees, 75 degrees, 80 degrees,85 degrees, 90 degrees, or any integer or non-integer value betweenthese values.

Thus, in some embodiments, the at least one third point on the surfaceof the object may be 45 to 90 degrees with respect to (or parallel orsubstantially/approximately perpendicular to) the plane of the image.Put differently, in some embodiments, the at least one third point onthe surface of the object may be at 45 to 90 degrees with respect to (orperpendicular or substantially/approximately perpendicular to) the planeof an optical lens of the imaging sensor. Thus, in some embodiments, thethird angle may be such that (e.g. the plane of) the surface of theobject at the third point and the plane of the image (or the plane ofthe optical lens of the imaging sensor) are perpendicular orsubstantially/approximately perpendicular. The edge spectrum mentionedearlier can thus be a representative spectrum for location(s) where theplane of the image (or the plane of the optical lens of the imagingsensor) is perpendicular or substantially/approximately perpendicular to(e.g. the plane of) the surface of the object.

In some embodiments, the second angle and the third angle may beswitched. For example, in some embodiments, the second angle referred toherein may be an angle in a range from 45 to 90 degrees and/or the thirdangle referred to herein may be about 0 degrees. In these embodiments,the difference between the spectrum of at least one second point on(e.g. the plane of) the surface of the object at the second angle withrespect to the plane of the image and the spectrum of at least one thirdpoint on (e.g. the plane of) the surface of the object at the thirdangle with respect to the plane of the image will have a different sign,e.g. a minus sign. Thus, in these embodiments, there is a minus sign insubsequent computations.

In some embodiments, the at least one second point on the surface of theobject may comprise at least one brightest point (or the averagebrightest points) on the surface of the object and/or the at least onethird point on the surface of the object may comprise at least onedimmest point (or the average dimmest points) on the surface of theobject. The at least one brightest point on the surface of the objectcan, for example, be at least one point on the surface of the objectwhere the second angle is about 0 degrees. The at least one dimmestpoint on the surface of the object can, for example, be at least onepoint on the surface of the object where the third angle is about 90degrees. In some embodiments where the object is the face of a subject,the at least one second point on the surface of the object may be atleast one second point on the surface of the tip of the nose on the faceof the subject and/or the at least one third point on the surface of theobject may be at least one third point on the surface of the edge of theface of the subject.

In some embodiments, the at least one second point on the surface of theobject may be identified by using landmark detection to detect at leastone second point on the surface of the object at the second angle withrespect to the plane of the image. Alternatively or in addition, the atleast one third point on the surface of the object may identified byusing landmark detection to detect at least one third point on thesurface of the object at the third angle with respect to the plane ofthe image. A person skilled in the art will be aware of establishedtechnology that can be used for landmark (e.g. facial landmark)detection in this way.

In some embodiments, a plurality of second points on the surface of theobject may be identified to detect at least one second point on thesurface of the object at the second angle with respect to the plane ofthe image and/or a plurality of third points on the surface of theobject may be identified to detect at least one third point on thesurface of the object at the third angle with respect to the plane ofthe image. In other words, measurements may be made over multiplelocations. Alternatively or in addition, filtering may be employed. Inthis way, a more robust characteristic curve may be determined.

Thus, in the manner described above, the at least one second point onthe surface of the object may be identified and the baseline spectrumacquired. Similarly, in this way, the at least one third point on thesurface of the object may identified and the edge spectrum acquired. Insome embodiments, the baseline spectrum can be acquired by the one ormore processors 102 of the apparatus 100 obtaining the baseline fromanother apparatus or device, by the one or more processors 102 of theapparatus 100 obtaining the baseline from a memory (such as the memoryof the apparatus 100 or another memory), or by the one or moreprocessors 102 of the apparatus 100 determining the baseline spectrum.Similarly, in some embodiments, the edge spectrum can be acquired by theone or more processors 102 of the apparatus 100 obtaining the baselinefrom another apparatus or device, by the one or more processors 102 ofthe apparatus 100 obtaining the baseline from a memory (such as thememory of the apparatus 100 or another memory), or by the one or moreprocessors 102 of the apparatus 100 determining the edge spectrum.

As an example, the baseline spectrum s_(baseline)[λ] and the edgespectrum s_(edge)[λ] may be determined by weighting of themulti-/hyperspectral spectra s[λ, x, y], where x is the x-coordinate inthe multi-/hyperspectral two-dimensional image, y is the y-coordinate inthe multi-/hyperspectral two-dimensional image and λ is the wavelengthindex indicative of the wavelength of light:

${s_{baseline}\lbrack\lambda\rbrack} = {\sum\limits_{{\forall x},y}{{w_{baseline}\left\lbrack {x,y} \right\rbrack} \cdot {s\left\lbrack {\lambda,x,y} \right\rbrack}}}$${s_{edge}\lbrack\lambda\rbrack} = {\sum\limits_{{\forall x},y}{{w_{edge}\left\lbrack {x,y} \right\rbrack} \cdot {s\left\lbrack {\lambda,x,y} \right\rbrack}}}$

In this example, the weight matrices sum up to unity:

${\sum\limits_{{\forall x},y}{w_{baseline}\left\lbrack {x,y} \right\rbrack}} = 1$${\sum\limits_{{\forall x},y}{w_{edge}\left\lbrack {x,y} \right\rbrack}} = 1$

The weight matrices of this example may be derived by transforming theoutput of a (e.g. facial) landmark detection using a fixed processingstep, ensuring that weights at the at least one second point (e.g. atthe tip of the nose) are high for w_(baseline)[x, y] and weights aroundthe at least one third point (e.g. around the edge of the face, such asthe jawline) are high for w_(edge)[x, y].

Following on from this example, the characteristic curve may bedetermined as follows:

s _(characteristic)[λ]=s _(edge)[λ]−s _(baseline)[λ].

FIG. 5 is an example schematic illustration of a hyperspectral image (onthe left) together with a hyperspectral map of a slice from that image(on the right). In this example, the hyperspectral image may be acalibrated reflectance image at a certain (arbitrary) wavelength. Thehyperspectral image is formed of a plurality of pixels, each pixelhaving a set of intensity values corresponding to light intensity valuesfor each of a plurality of wavelengths of light. The arrow 400 in thehyperspectral image indicates the location of the hyperspectral slice.That is, the arrow 400 in the image indicates the line along which acorresponding hyperspectral cube (x, y, wavelength) is sliced.

As illustrated in FIG. 5, in this example, the at least one second pointon the surface of the object at the second angle with respect to theplane of the image referred to herein comprises a point on the surfaceof the face (which is the object in this example) that is close to thenose. This is a point where the skin is (e.g. substantially orapproximately) parallel to the plane of the image or to the plane of anoptical lens of the imaging sensor. This point may also be referred toas the starting point of the hyperspectral slice. As also illustrated inFIG. 5, in this example, the at least one third point on the surface ofthe object at the third angle with respect to the plane of the imagereferred to herein comprises a point on the surface of the face (whichis the object in this example) at the edge of the face (just below theear). This is a point where the skin is (e.g. substantially orapproximately) perpendicular to the plane of the image or to the planeof an optical lens of the imaging sensor. This point may also bereferred to as the end point of the hyperspectral slice.

The hyperspectral map of the slice from the image indicates the distance(or, more specifically, the pixel distance) from the starting point ofthe hyperspectral slice (which is illustrated on the vertical axis)versus the wavelength (λ) index (on the horizontal axis). In thisexample, the wavelength index covers wavelengths in a range from 428 to1063 nm. The starting point of the hyperspectral slice is indicated byzero. The hyperspectral map of the slice from the image shows the lightintensity values for each of the plurality of wavelengths of light atthe different distances from the starting point. The whiter the pixel,the higher the light intensity value and thus the higher thereflectance. As can be seen from the hyperspectral map of the slice fromthe image, the way that the individual wavelengths begin to differ fromthe starting point (at zero) is different. This spectral dependency as afunction of distance, and thereby angle, becomes clearer after theactual difference from a first row is observed.

FIG. 6 illustrates the change in a calibrated hyperspectral image as afunction of distance (or, more specifically, the pixel distance) fromthe starting point of the hyperspectral slice and as a function ofwavelength (λ) index. More specifically, in FIG. 6, the relativereflectance is illustrated as a function of pixel position and λ index.The λ index covers wavelengths in a range from 428 to 1063 nm. Thestarting point of the hyperspectral slice is indicated by zero. Therelative reflectance is the relative reflectance with respect to thestarting point. As illustrated in FIG. 6, as the distance from thestarting point of the hyperspectral slice increases, a more and moredistinct pattern can be observed.

FIG. 7 is an example of a characteristic curve. In this example, thecharacteristic curve that is illustrated is a characteristic curve atthe maximum (angular) distance from the starting point of thehyperspectral slice. FIG. 7 illustrates an unfiltered version of thecharacteristic curve 500 and a filtered version of the characteristiccurve 502. The horizontal axis in FIG. 7 shows the wavelength (λ) index,which in this example indexes wavelengths non-uniformly sampled in arange from 428 to 1063 nm. The vertical axis in FIG. 7 shows thedifference in spectrum compared to original location or, morespecifically, the amplitude of the characteristic curve. That is, thevertical axis shows the difference between a spectrum of the at leastone second point on the surface of the object at the second angle withrespect to the plane of the image (or the plane of an optical lens ofthe imaging sensor) and the spectrum of the at least one third point onthe surface of the object at the third angle with respect to the planeof the image (or the plane of an optical lens of the imaging sensor).The characteristic curve indicates how the spectrum changes as afunction of distance and thereby angle.

In some embodiments, the characteristic curve referred to herein may bepredetermined (e.g. pre-calculated) using at least one othermulti-/hyperspectral two-dimensional image of the same type of object.For example, in embodiments where the object is the skin of a subject,the characteristic curve referred to herein may be predetermined (e.g.pre-calculated) using at least one other multi-/hyperspectraltwo-dimensional image of the skin of the subject and/or at least oneother multi-/hyperspectral two-dimensional image of the skin of one ormore other subjects, e.g. with different skin types. In some of theseembodiments, the characteristic curve referred to herein may be storedin a memory (e.g. a memory 106 of the apparatus or another memory), suchas in the form of a table of values. In some embodiments, the type ofobject (e.g. the skin type where the object is skin) may first beestablished based on the spectrum and the type of object may then beindexed in the memory, such as in the form of a table. In some of theseembodiments, each type of object may have a characteristic curve that ispredetermined (e.g. pre-calculated).

In other embodiments, the characteristic curve referred to herein may bedetermined (e.g. calculated) using the multi-/hyperspectraltwo-dimensional image of the object. For example, in embodiments wherethe object is the skin of a subject, the characteristic curve referredto herein may be determined (e.g. calculated) using themulti-/hyperspectral two-dimensional image of the skin of the subject.Thus, in some embodiments, the characteristic curve can be determinedbased on the actual image data.

In some embodiments, the characteristic curve described herein maycomprise a set of characteristic curves for a respective set of secondand third angles. In some embodiments, the characteristic curvedescribed herein may be selected from a set of characteristic curves fora respective set of second and third angles. In these embodiments, oncean image is acquired, a characteristic curve for the object in the imagemay be selected from a set of characteristic curves.

As mentioned earlier, in some embodiments, the characteristic curve forthe object may be a predetermined characteristic curve stored in amemory (e.g. a memory 106 of the apparatus 100 or any other memory),e.g. in the form of a look-up table. Thus, in some embodiments involvinga set of characteristic curves, the set of characteristic curves may bea predetermined set of characteristic curves stored in a memory (e.g. amemory 106 of the apparatus 100 or any other memory), e.g. in the formof a look-up table. In some embodiments, the set of characteristiccurves may have been determined upfront (e.g. in a lab setting). In someembodiments, the set of characteristic curves may be determined usingmachine learning or deep learning. Thus, in some embodiments that employmachine learning or deep learning, a set of characteristic curves as afunction of angle may be used, rather than a single characteristiccurve. In some embodiments involving deep learning, the deep learningmay be applied by feeding the training difference spectra andcorresponding angles.

Multiple characteristics curves can be beneficial for different objects,such as for different people. For example, different people can havedifferent skin types that may each have different characteristic curvesso a suitable characteristic curve for a certain skin type may beselected from a set of characteristic curves. Thus, in some embodiments,the set of characteristic curves may comprise characteristic curves fordifferent objects. For example, where the object is a person, the set ofcharacteristic curves may comprise a characteristic curve for each of aplurality of different skin types (e.g. classified according to theFitzpatrick Scale). Thus, the characteristic curve may be selected fromthe set of characteristic curves according to the skin type of theperson. That is, the characteristic curve that corresponds to the skintype of the person may be selected. In this way, the first angledetermination or the applied correction can be more accurate.

In some embodiments, the skin type of the person may be determined byway of any of the existing skin type determination techniques of which aperson skilled in the art will be aware. In other embodiments, a user(e.g. the person or another user) of the apparatus 100 may input theskin type of the person, e.g. via a communications interface 110 of theapparatus 100. Although skin type is used as an example, thecharacteristic profile may be selected based on any other properties ofthe object.

In some embodiments, the one or more processors 102 of the apparatus 100described herein may be configured to, for at least one pixel of theplurality of pixels corresponding to at least one other first point onthe surface of the object, compare the set of intensity values for saidat least one pixel to the characteristic curve for the object todetermine a measure of similarity of the set of intensity values to theobtained characteristic curve. In these embodiments, the one or moreprocessors 102 of the apparatus 100 described herein may also beconfigured to estimate at least one other first angle of the at leastone other first point on the surface of the object corresponding to saidat least one pixel from the determined measure of similarity. In some ofthese embodiments, the one or more processors 102 of the apparatus 100may also be configured to derive an angular map comprising the estimatedfirst angle and the estimated at least one other first angle. Thus,according to some embodiments, a first angle can be estimated for morethan one point and an angular map can then be derived.

FIG. 8 illustrates an example of a derived angular map. In this example,the object is a face of a subject. The eyes and area around the eyes areexcluded from the map for privacy reasons. In FIG. 8, the x and y axison the left denote the pixel position and the vertical axis on the rightdenotes the estimated angle (in degrees). In the derived angular mapillustrated in FIG. 8, the darkest parts of the angular map represent atleast one second point on the surface of the object at about 0 degreeswith respect to (or parallel or substantially/approximately parallel to)the plane of the image or to the plane of an optical lens of the imagingsensor. Similarly, in the angular map illustrated in FIG. 8, thelightest parts of the angular map represent at least one third point onthe surface of the object at about 90 degrees with respect to (orperpendicular or substantially/approximately perpendicular to) the planeof the image or to the plane of an optical lens of the imaging sensor.In some embodiments where an angular map is derived, the regressioncoefficient ‘a’ described earlier can be calibrated (e.g. scaled and/orclipped) to represent the actual angle using a single global gainparameter, e.g. such that the angle is close to 90 degrees for areasclose to the edge.

In some embodiments in which an angular map is derived, the one or moreprocessors 102 of the apparatus 100 described herein may be configuredto estimate a depth map using the derived angular map. Thus, in someembodiments, the angular map may be converted to a 3D image. Forexample, the angular map may be converted to a depth map (or 3D image)by starting from the location where the angle is substantially orapproximately 0 degrees (i.e. at a baseline location) and expandingoutwards estimating how far adjacent points are translated in depth as afunction of angle. Alternatively, in other embodiments, the angular mapmay be directly employed in the spectral decomposition process.

In some embodiments in which the object is skin, the one or moreprocessors 102 of the apparatus 100 described herein may be configuredto determine a concentration of chromophores in the skin from themulti-/hyperspectral two-dimensional image of the skin using theestimated first angle or from the multi-/hyperspectral two-dimensionalimage with the correction applied. A person skilled in the art will beaware of techniques that can be used to determine a concentration ofchromophores in the skin.

However, one example is a decomposition algorithm, which consists of amodel function f( ). The model function f( )describes the theoretical(reflectance or absorbance) spectrum as a function of frequency for agiven vector of chromophore concentrations c. Hence, f(c) maps towavelength λ. Then, as an example, for each pixel position in themulti-/hyperspectral two-dimensional image, the following least squareserror is minimized, using non-linear least squares optimization:

e[x, y]=Σ_(∀λ) |s[λ, x, y]−ƒ(c)|²,

where s[λ, x, y] denotes the multi-/hyperspectral spectra, x is thex-coordinate in the multi-/hyperspectral two-dimensional image, y is they-coordinate in the multi-/hyperspectral two-dimensional image, and λ isthe wavelength index indicative of the wavelength of light.

This results in a vector of chromophore concentrations c that bestmatches the input spectrum. The input spectrum is the set of intensityvalues for said at least one pixel described earlier, where the set ofintensity values correspond to the light intensity values for each ofthe plurality of wavelengths λ of light. More advanced models mayincorporate the angle, meaning that the function f( )not only has thechromophore concentrations c as input, but also the angle a. Thisresults in minimization of the following least squares error:

e[x, y]=Σ_(∀λ) |s[λ, x, y]−ƒ(c, a[x, y])|²,

where s[λ, x, y] denotes the multi-/hyperspectral spectra, x is thex-coordinate in the multi-/hyperspectral two-dimensional image, y is they-coordinate in the multi-/hyperspectral two-dimensional image, and λ isthe wavelength index indicative of the wavelength of light.

There is also provided a computer program product comprising a computerreadable medium. The computer readable medium has a computer readablecode embodied therein. The computer readable code is configured suchthat, on execution by a suitable computer or processor, the computer orprocessor is caused to perform the method described herein. The computerreadable medium may be, for example, any entity or device capable ofcarrying the computer program product. For example, the computerreadable medium may include a data storage, such as a ROM (such as aCD-ROM or a semiconductor ROM) or a magnetic recording medium (such as ahard disk). Furthermore, the computer readable medium may be atransmissible carrier, such as an electric or optical signal, which maybe conveyed via electric or optical cable or by radio or other means.When the computer program product is embodied in such a signal, thecomputer readable medium may be constituted by such a cable or otherdevice or means. Alternatively, the computer readable medium may be anintegrated circuit in which the computer program product is embedded,the integrated circuit being adapted to perform, or used in theperformance of, the method described herein.

There is thus provided herein an apparatus 100, method 200, and computerprogram product for estimating a first angle of a first point on asurface of an object from a multi-/hyperspectral two-dimensional imageof the object at respective wavelengths or in applying a correction tothe multi-/hyperspectral two-dimensional image, which addresses thelimitations associated with the existing techniques. There is alsoprovided herein an apparatus, method 300, and computer program productfor determining a characteristic curve for use in estimating a firstangle of a first point on a surface of an object from amulti-/hyperspectral two-dimensional image of the object at respectivewavelengths or in applying a correction to the multi-/hyperspectraltwo-dimensional image.

Variations to the disclosed embodiments can be understood and effectedby those skilled in the art in practicing the principles and techniquesdescribed herein, from a study of the drawings, the disclosure and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. A single processor or other unit may fulfil thefunctions of several items recited in the claims. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage. A computer program may be stored or distributed on a suitablemedium, such as an optical storage medium or a solid-state mediumsupplied together with or as part of other hardware, but may also bedistributed in other forms, such as via the Internet or other wired orwireless telecommunication systems. Any reference signs in the claimsshould not be construed as limiting the scope.

1. An apparatus for estimating a first angle of a first point on asurface of an object from a multi-/hyperspectral two-dimensional imageof the object at respective wavelengths or applying a correction to themulti-/hyperspectral two-dimensional image, the apparatus comprising oneor more processors configured to: acquire a multi-/hyperspectraltwo-dimensional image of an object at respective wavelengths, whereinthe multi-/hyperspectral two-dimensional image at the respectivewavelengths is formed of a plurality of pixels, each pixel having a setof intensity values corresponding to light intensity values for each ofa plurality of wavelengths of light; for at least one pixel of theplurality of pixels corresponding to a first point on a surface of theobject, compare the set of intensity values for said at least one pixelto a characteristic curve for the object to determine a measure ofsimilarity of the set of intensity values to the obtained characteristiccurve; and estimate a first angle of the first point on the surface ofthe object corresponding to said at least one pixel from the determinedmeasure of similarity, wherein the first angle is with respect to aplane of the image, or apply a correction to the multi-/hyperspectraltwo-dimensional image at the first point on the surface of the objectusing the determined measure of similarity, wherein the characteristiccurve characterizes how the object reflects or absorbs light as afunction of angle and is indicative of a difference between a spectrumof at least one second point on the surface of the object at a secondangle with respect to the plane of the image and a spectrum of at leastone third point on the surface of the object at a third angle withrespect to the plane of the image, wherein the third angle is a knownangle that is different from from the second angle.
 2. The apparatus asclaimed in claim 1, wherein: the characteristic curve is indicative of adifference between an average spectrum of at least two second points onthe surface of the object at the second angle with respect to the planeof the image and a spectrum of at least two third points on the surfaceof the object at the third angle with respect to the plane of the image.3. The apparatus as claimed in claim 1, wherein: the second angle isabout 0 degrees; and/or the third angle is an angle in a range from 45to 90 degrees.
 4. The apparatus as claimed in claim 1, wherein: the atleast one second point on the surface of the object comprises at leastone brightest point on the surface of the object; and/or the at leastone third point on the surface of the object comprises at least onedimmest point on the surface of the object.
 5. The apparatus as claimedin claim 1, wherein: the at least one second point on the surface of theobject is identified by using landmark detection to detect the at leastone second point on the surface of the object at the second angle withrespect to the plane of the image; and/or the at least one third pointon the surface of the object is identified by using landmark detectionto detect the at least one third point on the surface of the object atthe third angle with respect to the plane of the image.
 6. The apparatusas claimed in claim 1, wherein: the characteristic curve ispredetermined using at least one other multi-/hyperspectraltwo-dimensional image of the same type of object; or the characteristiccurve is determined using the multi-/hyperspectral two-dimensional imageof the object.
 7. The apparatus as claimed in claim 1, wherein the oneor more processors are further configured to: for at least one otherpixel of the plurality of pixels corresponding to at least one otherfirst point on the surface of the object, compare the set of intensityvalues for said at least one other pixel to the characteristic curve forthe object to determine another measure of similarity of the set ofintensity values to the obtained characteristic curve; and estimate atleast one other first angle of the at least one other first point on thesurface of the object corresponding to said at least one other pixelfrom the determined another measure of similarity; and derive an angularmap comprising the estimated first angle and the estimated at least oneother first angle.
 8. The apparatus as claimed in claim 1, wherein: thecharacteristic curve for the object is selected from a set ofcharacteristic curves for a respective set of second and third angles.9. The apparatus as claimed in claim 1, wherein: the spectrum of the atleast one second point comprises (1) a reflectance spectrum indicativeof the portion of light reflected from the object at the at least onesecond point on the surface of the object at the second angle withrespect to the plane of the image or (2) an absorbance spectrumindicative of the portion of light absorbed by the object at the atleast one second point on the surface of the object at the second anglewith respect to the plane of the image; and/or the spectrum of the atleast one third point comprises (1) a reflectance spectrum indicative ofthe portion of light reflected from the object at the at least one thirdpoint on the surface of the object at the third angle with respect tothe plane of the image or (2) an absorbance spectrum indicative of theportion of light absorbed by the object by the object at the at leastone third point on the surface of the object at the third angle withrespect to the plane of the image.
 10. The apparatus as claimed in claim1, wherein the object is skin and the one or more processors areconfigured to: determine a concentration of chromophores in the skinfrom the multi-/hyperspectral two-dimensional image of the skin usingthe estimated first angle or from the multi-/hyperspectraltwo-dimensional image with the correction applied.
 11. A method forestimating a first angle of a first point on a surface of an object froma multi-/hyperspectral two-dimensional image of the object at respectivewavelengths or applying a correction to the multi-/hyperspectraltwo-dimensional image, the method comprising: acquiring amulti-/hyperspectral two-dimensional image of an object at respectivewavelengths, wherein the multi-/hyperspectral two-dimensional image atthe respective wavelengths is formed of a plurality of pixels, eachpixel having a set of intensity values corresponding to light intensityvalues for each of a plurality of wavelengths of light; for at least onepixel of the plurality of pixels corresponding to a first point on asurface of the object, comparing the set of intensity values for said atleast one pixel to a characteristic curve for the object to determine ameasure of similarity of the set of intensity values to the obtainedcharacteristic curve; and estimating a first angle of the first point onthe surface of the object corresponding to said at least one pixel fromthe determined measure of similarity, wherein the first angle is withrespect to a plane of the image, or apply a correction to themulti-/hyperspectral two-dimensional image at the first point on thesurface of the object using the determined measure of similarity,wherein the characteristic curve characterizes how the object reflectsor absorbs light as a function of angle and is indicative of adifference between a spectrum of at least one second point on thesurface of the object at a second angle with respect to the plane of theimage and a spectrum of at least one third point on the surface of theobject at a third angle with respect to the plane of the image, whereinthe third angle is a known angle that is different from from the secondangle.
 12. An apparatus for determining a characteristic curve for usein estimating a first angle of a first point on a surface of an objectfrom a multi-/hyperspectral two-dimensional image of the object atrespective wavelengths, wherein the first angle is with respect to aplane of the image, or in applying a correction to themulti-/hyperspectral two-dimensional image, the apparatus comprising oneor more processors configured to: acquire a first spectrum of at leastone second point on the surface of the object at a second angle withrespect to a plane of the image; acquire a second spectrum of at leastone third point on the surface of the object at a third angle withrespect to the plane of the image, wherein the third angle is a knownangle that is different from or substantially different from the secondangle; and determine the characteristic curve as a difference betweenthe first spectrum and the second spectrum, wherein the characteristiccurve characterizes how the object reflects or absorbs light as afunction of angle.
 13. A method for determining a characteristic curvefor use in estimating a first angle of a first point on a surface of anobject from a multi-/hyperspectral two-dimensional image of the objectat respective wavelengths, wherein the first angle is with respect to aplane of the image, or in applying a correction to themulti-/hyperspectral two-dimensional image, the method comprising:acquiring a first spectrum of at least one second point on the surfaceof the object at a second angle with respect to a plane of the image;acquiring a second spectrum of at least one third point on the surfaceof the object at a third angle with respect to the plane of the image,wherein the third angle is a known angle that is different from orsubstantially different from the second angle; and determining thecharacteristic curve as a difference between the first spectrum and thesecond spectrum, wherein the characteristic curve characterizes how theobject reflects or absorbs light as a function of angle.
 14. Anon-transitory computer readable medium, the computer readable mediumhaving a computer readable code embodied therein, the computer readablecode being configured such that, on execution by a suitable computer orprocessor, the computer or processor is caused to perform the method asclaimed in claim 11.